Site crawlers and web rating security applications may be used to determine a rating for websites accessible via the Internet. Site crawlers may be used to gather information (e.g., the content of a website) by visiting the websites. This information may then be used by a web rating security application to, for example, determine a reputation for each site visited. This reputation may then be used to determine whether the particular site is a safe site or is associated with a malicious activity.
These applications, however, are not able to address several online security risks. For example, site crawlers cannot detect content and links, such as user profiles and online interactive experiences within social networks, that are only accessible by entering a password. Additionally, site crawlers traditionally find content via crawling a site but do not know what link the users of the site click on and the contents of those links. Many hackers have also learned how to circumvent web rating security applications by cloaking the payload or destination associated with the link behind multiple redirects and shortened Uniform Resource Locators (URLs).